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Effects of Instructional Setting and Approach 
in Compensatory Education: A Statewide Analysis 

Abstract 

-i ■ ' ■ 

The present study grew out of an^ increased awareness of iypformation 
needs in making curricular decisions at the /school level. Data were 
coded from evaluation reports for some 36o Chapter 1 projects over a 
three-year period. Among, other things, different project settings and 
instructional approaches were examined in the study. Results suggest a 
discernible trend favoring the pull-out setting. While there was some 
interaction between project setting and grade level, such effects were 
generally negligible. Overall, an eclectic instructional approach (e.g., 
a prescribed system supplemented by teacher-made materials) was as 
effective as, if not more so than, other approaches. Some interaction 
was found between -instructional approach and grade level. 



4 



Ef ftects of Instructional Setting and 
Approach in Compensatory Education: A Statewide Analysis 



Kim Onn Yap 
Northwest Regional Educational Laboratory 



A paper presented at the annual meeting of the American Educational 
Research Association, Montreal, April 11-15, 1983. 



ERIC . , ^ 



Effects of Instructional Setting and 
;^proach in Compensatory Education: A Statewide Analysis 

INTRODUCTION 

Chapter 1 represents one of the largest federal investments in 

m 

education for disadvantaged youth in the nation. Typically, Chapter 1 
projects provide remedial instructional services in basic skills areas such 
as reading and mathematics. In a majority of the projects, the services 
were provided in a pull-out setting (Stonehill and Anderson, 1982) . 
Instructional approaches used in these projects are often described as 
"eclectic" involving the use of teacher-made as well as commercially 
packaged materials. The adoption or adaptation of instructional settings 
and approaches is often a matter of logistical convenienfce or fiscal 
necessity rather than an outcome of careful deliberations of benefits and 
impact. While there has been^, little direct evidence to indicated the 
efficacy, or the lack of it, of the pull-out setting, negative perceptions 
have been expressed by scwie researchers. Harnischfeger (1980), for 
example, described two major shortcomings of the pull-out setting. 
First, the pull-out teachers have little contact with the classroom 
teachers^ As a result, there is little integration of the pull-out with 
the classroom instruction and the classroom teacher loses closf contact 
with the progress of students who are pulled-out. Secondly, if students 
receive reading instruction in a ^pull-out program, usually they are 
pulled-out of classroom reading instruction. Consequently, pull-out 

r 

students do not receive extra learning opportunity and often receive less 
reading time than the rest of the class due to increased transition time. 
It may be hypothesized, on the other hand, that/since the pull-out 
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setting often meana smaller class size and generally provides greater 
instructional Support to students, it could be eatpected to produce greater 
achievement (Tobias, 1982).. Smith and Glass (1980) for instance, found 
that reducing class size h^ad benefical effects both on cognitive and 
affective outcomes and on the teaching process itself. 

It has been reported that major programs in compensatory education that 
appear jd/fferent do not seem to have differential impact upon the 
participants. In a^rev^eW of effects of different curricula in early 
childhood eudcation^ Weikart (1981) found that the ex^)ected major 
differences between curricula simply were not obtained. The author 
suggested that the basic issues in successful programming had to do with 
the quality of program iiif5>lementatioa rather t^an the philosopl^y of 
curriculum selected. 

In much the same vein, Tobias (1982^ argued that external differences 
between instructional treatments, whether they are educational media/, 
methods of or^aniziftg classroom, or technological devices, are iii«)ortant 
only in terms of the degree to which* they influence the student's cognitive 
activities while engaged by the instructional content. External 
differences in instructional arrangMents that lead to similar cognitive 
processing will result in comparable achievement irrespective of 
superficial differtence between methods. The author suggested that, 
generally, any instructional method or procedure which induced students to 
spend more time on ta^^ than a comparison mode would result in higher 
achiveveroent. 

TolDias. (1976, 1981) hypothesized that students with low prior 
achievement required a good deal of instructidnal support to accomplish 



objectives. Conversely, those higher in prior achievement needed little 

♦ 

support. Prior achievement was defined by students' pretest scores. 
Instructional support was defined as the assistance' given the learner in 
organizing the instructional content ^ maintaining student attention ^ 
eliciting responses, providing feedback on tiie re^nse, and so on. 

Results from enqpirical work on the in?>act of instructional settings on 
student achievement in compensatoBy education have been equivocal or 
ambiguous. In the Instructional Dimensions Study (IDS), Cooley and 
Leinhardt (1980) found that pull-out instruction was related inversely with 
reading achievement. However, in a reanalysis of the same data involving 
only pull-out students a positive relationship was evident between pull-out 
and posttest achieveatent in reading (Leinhardt and Pallay, 1982). 
Incidentally, teachers involved in the IDS project expressed mixed opinions 
about the pull-out setting. Pull-out was said to be negative because it 
was a managerial headache, demoralized the children, and disrupted the 
day. It was said to be positive because more intensive remediation could 
be given to children who needed it. In_an evaluation of the Emergency 
Schoold Aid Act, Coulson et al. (1977) found evidence suggesting that 
pull-out was associated with lower achievement. The authors reported that 
a change from receiving no pull-out instruction in reading to receiving one 
half reading instruction in pull-out was associated with a reduction in 
reading achievement. Although the effect was small, the finding does not 
support the noticm that pull-out is a positive practice. 

Leinhardt and Pallay (1982) reported empirical work that demonstrated 
success, failure, and no difference in a variety of instructional 



settings. The authors concluded that setting was not a clearly 
overwhelming variable in and of itself. However ^ it was worth 
investigating and speculating how setting influenced features of 
instruction that in turn influenced student learning time. 

Findings of no significant differences between 'the pull-out and regular 
classroom setting are consistant with Dahllof 's (1971) assertion that 
achievement cannot be regarded as a direct outcome of the grouping 
arrangement. As Leinhardt and Pallay (1982) pointed out, settling does not ^ 
eliminate or guarantee the presence of effective instructional practices. 
Most of the important variables can occur in roost settings, 

>« 

TH? STUUY 

V 

The Hawaii Department of Education has systematically gathered Title 1 
data since the inception of evaluation requirements in the Title 1 
legislation more than a decade ago. Little, however, was don^ to integrate 
the data either on a longitudinal or cross-sectional basis to address 
questions relating to statewide program activities. It was a widely shared 
perception within the Department that evaluation data gathered during the 
past several years had not been put to maximum .use. There wa^ a need to 
find out what types of data had been collected and how such data could be 
used to address subtantive issues relating to Hawaii's overall Title 1 
effort. The study grew out of an increased awareness of information needs 
in making curricular decisions at the school level (David, 1981; Alkin,^et 
al. , 1982). It was conceived jointly by an external researcher and 

i' 



ERIC 



internal project staff and took the form of a secondary ai)Alysis (Burstein, . 
1978) of evaluation data gathered over a period of three years. The study 
addressed the following research questions: 

1. What are the effects of different instructional settings on the 
achievement of students participating in Chi«)ter 1 projects? 

2. What are the effects of different instructional approaches on the 
achievement of students participating in Cha«)ter 1 projects? 

The primary audience of the project was to be the Hawaii state office 
personnel (e.g., evaluation and compensatory education staff). It becarc 
clear, however, that district coordinators and Title 1 school administrators 
as well as the teaching s|aff could also benefit form the project. Plans 
were then made to disseminate the results to a much larger audience than 

the state office staff. 

»to experimental manipulation was used in the study. Data were coded 

from evaluation reports for some 300 Chapter 1 projects over a three-year 

petiod. Among other things, three project settings and three instructional 

approaches were examined in the study: 

I. Project Setting 

A. Pull-out. This involved the pulling out of participating students 
from the regular classroom to receive remedial instructional 
services from the Chapter 1 teacher or teacher aide in a small 
group setting. 



Note: Federal legislation changed Title 1 to Chapter 1 in 1981. 



B. Regular classroom ^ The setting required the Chapter 1 teacher or 
aide to intervene directly in the regular clasarooni. The teacher 
or aide worked with project students while instruction was being 
provided to non-project students by the regular classroon teacher 
at the sane time. 

C. Combination , Tt.is involved the provision of instructional 
services in a combination of pull-out and regular classroom 

^ settings. 
II, Instructional Approach 

A, Instructional system . This consisted of the use of a prescribed . 
system with a set of commercially packaged materials along with 
protocols for their use, 

B, Materials only . This approach allowed the Chapter 1 teacher or 
aide to use materials, both domaercially packaged and 
teacher-made, in ways she/he saw fit without having to follow 
pre&cribed protocols, 

C, Combination, This consisted of the adaptation of a prescribed 
instructional system along with associated materials and the use 
of a miscellany of other materials, commercial or teacher-iisaae, to 
supplement the prescribe^ system, 

PROCEDURE 

The investigation was carried out in cooperation with the state 
evaluation and compensatory education staff. The involvement of 
compensatory education staff was particularly crucial with respect to 
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formulating and prioritizing questions to be addressed. The scope of the 
study depended in a large measure on feedbaclt provided by the compensatory 
education staff. * 
De sign 

The study was conceptualized in early 1981 when data-^se became an area 
of interest to both state and local district staff in Hawaii. The state 
evaluation staff, in particular, perceived a need to build a data base by 
pooling data presented in evaluation reports for past school years and to 
use the data base to address statewide issues relating to Chapter 1. 
Several discussions were held during the ©arl/ months of 1981 Mhich 
resulted in the delineation of the following design elements: 

1. Data sources would be limited to evaluation reports prepared by 
the external evaluator and related documents (e.g.f state 
directories and welfare reports). 

2. The study would cover data, for ther* 1978-81 school years. 

3. Schools would be used as the unit of analysis. Data were to be 
aggregated across graces for each school. (This decision was 
later relaxed to accommodate grade-by-grade analysis of 
achievement data.) 

A preliminary list of variables of intere*-was compiled by the state 
evaluation staff on "ffie basis of a review of several school-level 
evaluation reports. The list was reviewed by the author and subsequently 
served as a basis for the development of a coding format* 

State evaluation and compensatory education staff were involved in 
formulating and prioritizing research questions for the study. Following 



s*v«r«l discussions, it was decided that while all variables identified . 
were of interest to the staff, the study would focus on questions "relating 
to the effects of various instructional approaches and project setting on 
student achievement. 
Data Coding 

Data coding was acomplished in three Eftiases. Preliminary coding of 
project and school information was conducted by research. assistants -at the 
laboratory in Portland. These data pertained essentially to information 
obtainable from the district-level reports. Such information items 
included school code, school enrollment, schoo^. type, grade levels covered 
by school, welfare status of attendance areas and overall student 
achievement. 

The second phase of data coding was conducted on-site in Hawaii with 
the -aid of graduate students hired by the compensatory education section. 
The coding covered some 300 plus school-level evaluation reports and 
included information obtainable only from the school-level reports. Such 
information included student grouping, diagnostic testing, inservice - 
training, project setting and instructional approach. With respect to the 
coding of instructional approaches a list of basal materials used in 
Chapter 1 projects was prepared by the compensatory education staff. The 
list was reviewed by language arts specialists at the state department of 

t 

education. Based on this review, the three primary categories of 
instructional approaches mentioned earlier were established. 

The third phase of data coding included grade-by-grade achievement data 
within each project. This was done following considerable discussion on 
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comparability of NCE gains across grades, resource constraints and the 
merits of having a grade-by-grade analysis of achievement data. The coding 
was acccHBplished by student helpers at the evaluation section. 

As would be expected/ a^number of problems were encountered in coding 
information from the evaluation reports, including: 

1. Missing data . Information was not provided in the district or 
school-level reports. For example, there were no data on staff 
PTE and absenteeism for the 1978-79 school year. No information 
on project setting and instructional approach was obtained for 
secondary projects for the 1978-79 and 1979-80 school years. 

2. Ambiguous narrative^ . Some {Project narratives in the school-level 
reports were difficult to interpret. For example, in some cases 
membership of parent advisory councils was not clearly described. 

3. Lack of discreteness . Many projects used composite project 

/ 

settings and student groupings. For instance, some narratives 
mentioned use of individual, small group and large group 
instruction with various grouping configurations. 
Most problems were resolved on the ijasis of the data coders' best 
judgment. In other cases, the data sheets were left blank. 
Data Analysis 

Con?>leted data sheets were mailed to the author at the laboratory in 
Portland. These were key-punched and quality control measures were taken 
to ensure accuracy of coding ^nd validity of the coded data. A few cases 
were discarded because of excessive missing data and some inaccuracies in 
coding were corrected. Data analyses were then performed on the "clean" 
data. 
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Initially^ means and standard deviations were computed for all 
variables of interest* These calcuj^tions were conductec^ separately for 
^ach of the three school years included in the study. Descriptive 
statistics were obtained on the selected variables by subject area (i.e., 
reading versus mathematics) and by school type (i.e., public versus 
private) . Due to the relatively small number of projects in private 
schools, subsequent analyses were confined to projects in public schools. 
(See Table 1 for list of selected variables.) 



Table 1 about here 

Correlational^iinalyses were then performed on samples of public school 
reading projects, again separately for each of the three school years. 
These analyses were conducted to identify factors which were related to 
student achievement. 

A third set of analyses consisted of analyses of covariance (ANCOVA's) 
of project NCE gains using per pupil cost and pretest NCE as covariates, 
the latter variables having been shown to be related to student 
achievement. The ANCOVA's were performed to assess f±L effects of project 
settings and instructional appi;oaches commonly used in Chapter 1 projects 
in Hawaii. Again, these analyses were performed separately for each of the 
three school years. The ANCOVA's were first performed on NCE gains for 
projects as a whole — grade level gains having been agregated across 
grades. Similar analyses wec^e then conducted separately for each grade. 



RESULTS 

Project Profile ^ 

Based on descriptive statistics provided in the analyse^r project 
profiles were developed for Chapter 1 projects included in the study • The 
data indicate that the average Chapter 1 reading project enrolled? 98-127 . 
students in five different grades and provided 96-99 houra of instruction. 
The project employed 3.4-4.9 FTE staff and produced an achievement gain of 
7-8 NCEs at a per pupil cost of $672-$712. The average math project 
enrolled 50-67 students in five different grades and provided 61-75 hours 
of instruction. The project employed 3.0-7.0 FTE staff and produced an 
achievement gain of 11 NCEs at a per pupil cost of $458-$488. 

A comparison of project profiles suggests that while reading project 
configurations (e.g.f project enrollment, duration, hours of instruciton 
and costs) fluctuated somewhat over the years, project impact in terms of 
NCE gains has ifemained relatively stable, ranging from a low of 7.40 for 
the 1979-80 school year to a high of 8.17 for the 1980-81 school year. 
Pretest status has increased from 18.7 NCEs (7 percentile) for the 1978-79 
school year \q 22.6 (10 percentile) for the 1980-81 school year. On the 
other hand, increasingly less money was spent on the average student. The 
per pupil cost declined from- $712 in 1978-79 to $6T3 in 1980-81. 

Configurations of* math projects also vaxfied somewhat over the years. 
Again, project impact in terms of NCE gains has remained relatively stable. 
Consistent with results obtained from national studies, NCE gains in math 
projects were larger than those found in the Reading projects. For all 
three school years included in the study, such gains have been 
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approximately 11 NCE points. Also, the pretest status o# project students 
increased from 21.6 NCEs (9 percentile) in 1978-79 to 23.99 (11 percentile) 

in 1980-81. '-^ 
Factors Relatted to Achievement 

For *th4 1978-79 school year, ten of the variables of interest were 
found CO be significantly (p<.05) related to student achievement. These 
included school type (-.30), grade levels covered by school (.25), project 
enrpllment (-.21), size of project staff (-•20), 8chool enrollment ^-•20), 
number of students tested (-.23), pretest NCE scores (-.30), posttest NCE 
scores (.48), percent of students making positive gains (.79) and per pupil 
cost (.37). The correlation coefficients which ranged from low to 
moderately high in magnitude suggest the following: 

1. Students in lower grade levels made greater NCE gains. 

2. Projects with smaller enrollments tended to produce greater NCE 
gains. 

3. Project with smaller project staff tended tjo produce greater NCE 
gains. 

4. Projects which were located in smaller schools tended to produced 
greater gains. 

5. Projects with higher funding levels tended to produce greater 
gains.. 

6. Projects with a lower pretest status tertded to end up with higher 
NCE gains. 

Taken together, the results indicate a project/school size effect. 

/ r 

That is, projects with small enrollments located in small schools were 
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shown to have favorable impact on student achievement. That students at 
lower grades tended to make greater NCE ^ains and that projects with a 
lower pretest status tended to end up with higher NCE gains are generally 
consistent with natipnal TI£RS results. 

For the 1979-80 school year,' 11 of the variables were shown to be 
significantly (p<.05) related to student achievement. These variables were 
school type (-.21), grade levels covered by project (.31), inservice 
training on evaluation (.20), size of project staff (-.24), school 
enrollment (-.24), size of school staff (-.25), pretest NCE scores (-•^8), 
posttest NCE scores (.36) , percent of student^ making positive gains (.82) , 
number of days absent (-.39) and per pupil cost (.22) . The correlation 
coefficients, ranging from low to moderately high, support the following 
inferences: 

1. Students in lower grade levels made greater NCE gains. 
2.. When project staff received inservice training on evaluation, 
project outcomes appeared to be more positive. 

3. Projects located in relatively small schools tended to produced 

^? 

greater NCE gains. 

s 

4. Projects with lower pretest status tended to end up with greater 
NCE gains. 

5. Absenteeism was shown to have an adverse effect on student ' 
achievement. 

6 J Projects with higher funding levels tended to produce greater NCE 
gains. 
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By and large, these findings represent a. replication of those obtained 

for the preceding school year. Two additional findings are, however, 

■ > .' ' ' 

noteworthy. One, when project staff received inservice training on 

evaluation, project outcomes appeared to be more positive. This may be due 

to better quality control measures being instituted (e.g., better test 

^dminiatration and scoring) as a result of the inservice, Twor number of 

days absent was found to be inversely related to NCas gains, This finding 

is consistent with results of recent studies conducted on time 6n task 

(£[arniBchfeger, 1980). 

For the 1980-81 school yearr nine of the vatiables included in the 

study were found to be significantly (p<.05) related to student achievement. 

These variables were school type (-.32) ^ grade levels covered by school 

{.25)^ project enrollment (-.26), school enrollment (-.35), size of school 

*taff (-.37), number of students tested {".H) , pretest NCE scores (-.33), 

posttest NCE scores (.58) and percent of atudents making positive gains 

(.37). Again, the correlation coefficients ranged from low to moderately 

high in magnitude and appeared to support the following interpretations: 

1. Students in lower g.ude levels made greater NCK gains. 

2. Project with smaller enrollments tended to produce greater NCE 
gains. 

3. Projectc located in smaller schools tended to produce greater NCE 
gains. 

4. Projects with a lower pretest status tended to end up with greater 

if 

NCE gains. . 

8 
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These findings are generally consistent with those obtained for the 
preceding school years — particularly so with respect to the favorable 
effed&s of small school/project size. A significant departure is the fact 
that per pupil cost was not shown to be related to student achieveinent. 
The absence of a relationship inay^ in part^ be due to the way in which cost 
was reported for the i9'80^1 school year. It. is noted that the standard 
deviation of per pupil cost for the 1980-8i school year nearly equaled the 
mean and was substantially larger than the standard deviations for the 
preceding school .years. 

Taking the results for the three school years into account b/ie readily 
discerns a generally consistent pattern. The pattern Suggests foui: 
replicable findings. One, students in lower grade levels tended to make 
greater NCE gains than those in higher grades.. Two, there was ^e 
evidence that smaller was better when it came to project and/or school 
size. There was an inverse relationship between project/school size and 
student achievement. Three, projects which started off with a lower 
pretest status generally ended up with greSiter NCE gains. Four, per pupil 
cost — probably a surrogate for other important program elements — was a 
potent predictor of student achievement. There appeared to be a positive 

i 

relationship between cost and NCE gain. 

It should be noted that factors which have been found to be 
significantly related to project isnpact do not necessarily "explain" why 
students in a project made NCE gains. Nd causal relations should be 
inferred from results of the correlational analysis. The findings, 
nonetheless, contribute to a more meaningful interpretation of NCE gains 
and better understanding of the complexity of project implementation. 
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Effects of Project Setting 

Reading . For the 1978-79 school year^ a majority of the reading 
projects for which^ information was made available provided instructional 
services in a pull-out setting. A small number provided instruction in the 
regular classroom. Some projects ^used a cc»nbination of settings (see Table 
2). Results of the analysis of covariance (with, effects of pretest and 
cost partialled out) suggest that "pull-out" represented an effective 
setting for providing Chapter 1 services. Students who received 
instruction in this setting performed as well as- if not better than, their 
counterparts in other settings, although the difference did not reach 
statistical significance at the .05 level. Specifically, these students as 
a group made, an NCE gain of 8.73 as compared with gains of 5.91 and 7.64 in 
"regular classroom" and "combination," respectively. 

The superiority of the pull-out setting was also evident in the 1979-80 
achievement data. Students who received instructional services in this 
setting made an NCE gain of 8.30. Students in "regular /classroom" and 
"combination" made gains of 5.08 NCEs and 6.18 NCEs, respectively. (See 
Table 3.) The difference, after adjusting for pretest and cost, was 
statistically non-significant at the .05 level, however. 

Projects which provided instruction in a pull-out setting during the 
1980-81 school year produced an average NCE gain of 9.73. This compared 
favorably with gains of 6.78 NCEs and 8.64 NCEs produced by "regular 
classroom" projects and "combination" projects, respectively. (See Table 
4.) Again, the difference did not reach statistical significance at the 
.135 level when effects of cost and pretest status were partialled! out. 
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Tables 2-4 about here 



Math. Math projects were combined for all three school years to yield 
a larger sample size for the analysis of covariance. Among projects for 
which information was made available ^ approximately one-half provided 
instruction in the regular classroom setting and the other half used the 
pull-out setting. (See Table 5.) Results showecl that students in the 
pull-out setting made substantially greater gain (13.33 NCEs) thdft their 
counterparts in the regular classroom setting (8.36 NCEs) , although the 
difference^ after adjusting for cost and pretest status, failed to reach 
statistical significance at the .05 level. 



Table 5 about here 



Interaction between grade level and project setting . Due to small 
sample sizes and missing data problems, no attein>t was made to ascertain 
interaction effects through a formal factorial analysis of covariance. 
Instead, actual and adjusted means were calculated for the respective 
settings separately for each grade level. 

Results for the 1978-79 reading projects, which included data for 
grades two through eight, showed that the effects of project settings were 
4>retty much "balanced" out when grade level was taken into consideration. 



Specifically r the rc^gular classroom setting was shovm to be particularly 
effective at grades four and seven. The pull-out setting appeared superior 
with sixth-and eighth-graders while combination of settings was apparently 
most effective with students in grades two^ three and five. The 
differences were^ however r not substantial in most cases. 

9 

For the 1979-80 reading projects^ the pull-out setting- was shown to be 
superior for grades three^ six and seven. A combination of "pull-out" and 
"regular classroom" appeared to have the most favorable impact on grades 
tyio, fourr five and eight. In no case was the regular classroom setting 
found to be superior (statistical significance aside) to the other settings. 

By far the 1980-81 results for reading projects appeared roost 
consistent with the overall trend described earlier when grade level was 
not taken into account. The full-out setting was shown to be most 
effective in eight out of 11 instances. A combination of "pull-out" and 
"regular classroom" was superior in the other three instances^ i.e.^ for 
grades twoy three and eight. Again^ in roost cases, the differences 
appeared to be not substantial. 

Results for the math projects were also generally consistent with the 
overall trend. In a majority (five out of seven) of instances, the 
pull-out setting was shown to be moat effective. The regular classroom 
setting appeared superior with students in grades five and seven. 

Sunwary . The overall analysis pr«>Juced a clearly discernible trend 
indicating superiority of the pull-out setting when grade level was not 
taxen into consideration. While there appeared to be some interaction 
between project setting and grade level (i.e., effects of a project setting 
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are somewhat dependent on the grade level in question) , such effects were . 
generally not substantial. For both reading and math projects, the overall 
trend geemed to favor the pull-out setting. 
Effect of Instructional Approach 

Reading . For the 1978-79 school year, information on instructional 
approaches was difficult to extract from the sciiool or district level 
evaluation reports. As a result, a majority of the 'projects were grouped 
under the "others" category (see Table 6) . Among projects for which 
information was obtained, most used a combination of instructional 
approaches, i.e., materials with a prescribed instructional system as well 
as materials -without such a system. Six- projects used materials with a 
prescribed instructional system and three used materials without a 
prescribed system. The results showed that a combination of approaches may 
well be superior to either approach. Students who received instruction in 
a combined approach made an average NCE gain of 10.39, comparing favorably 
with gains of 8.32 NCEs and 4.83 NCEs under a prescribed instructional 
system and under materials only, respectively. 

Results for the 1979-80 school year revealed a similar trend. The 
combination approach was shown to be superior, producing an NCE gain of 
9.37. Achievement gains for "instructional systems" and "instructional 
materials only" were 6.50 NCEs and 5.53 NCEs, respectively. After 
adjusting for pretest and per pupil cost, differences among the 
instructional approaches were shown to be statistically significant aC the 
.05 level. (See Table 7.) 

For the 1980-81 school year^ the three approaches (i»e.f instructional 



systems, instructional ma^ials only, combination) appeared to have 
produced highly simila^.^ach^evement gains, ranging from 9.23 NCEs to 10.07 
NCEs. These differences were not only statistically non-significant at th. 
J35 level but also seemed negligible in magnitude. The trend favoring the 
combination approach was not evident in the 1980-81 data. (See Table 8.) 



Tables 6-8 about here 

Math. Eleven projects which provided mathematics instruction to 
participants used the combination approach. Thirteen projects used 
approaches which had to be classified as "others." No relevant information 
was obtained for the other math projects. Results of the analysis showed 
that the combination and the "others" approaches produced highly similar 
NCE gains— 10.32 and 10.56^ respectively. The difference was statistically 
non-significant and minuscule. (See Table 9.) ^ 



Table 9 about here 

% 

% 

Interaction between grade level and instructional approach . Due to 
small sample sizes and missing data problems ^ no attempt was made to 
ascertain interaction effects through a formal factorial analysis of 



covariance. Instead, actual and adjusted means were calculated for the 
respective instructional approaches in reading projects separagely for each 

grade level. 

s 

Results for the 1978-79 school year showed that the combination 
approach was superior for grades two and five*- Students in grades three, 
four, six and eight appeared to do well with "instxuctional materials only." 
Seventh-grade youngsters who received instruction under a prescribed system 
appeared to perform better than their counterparts under "instructional 
materials only" or the combination approacdi. 

For the 1979-80 school year, the combination approach was found to have 
produced the greatest NCE gains for all grade levels included in the 
analysis except the eighth-grade. All instructional approaches appeared to 
. be equally effective for the eighth-graders, the differences in NCE gains 
being negligible in magnitude. 

- The superiority of the coabination approach appeared less evident in 
the 1980-81 data. For three of the grade levels (four, five and six), this 
approach was shown to be more effective than other approach^».^n the 
other hand, a prescribed ins^uctional system seemed to work better with 
youngsters in grade two, seven and eight. -Instructional materials only" 
was shown to be roost effective with third-graders. 

Sunnary . Although the trend was not very clear, the analysis seemed to 
have produced some evidence favoring the combination approach. This 
approach was shown to have produced the greatest NCE gains for the 1978-79 
and 1979-80 school years. All three approaches were found to be similarly 
effective for the 1980-81 school year. When grade level effects were taken 



into consideration, the trend became less evident, as would be expected. 
Overall, however, the combination approach was shown to be a& effective as, 
if not more so than the other approaches in helping Title 1 youngsters. 

DISCUSSION AMD CONCLUSIONS / . 

Among project characteristics included in the study, previous 
investigations have shown class" size (Glass and Smith, 1978; Glass and 
Smith, 1979; Smith and Glass, 1980), learning time (DeValuXt, et al., 1977; 
Pislfer, et al. , 1978; Harnischfeger and Wiliy, 1978; Stallings, 1980; 
Harnischfeger, 1980), and expenditures (Polley, 1976) to be related to 
student achievement. Attendance rate, once socioeconomic factors were 
accounted for, was found iiot to be asso&iated with achievement (Polley, 
1976). Results of the present study generally lend support to some of the 
previous findings even -though the latter were obtained with general student 
populations rather than Chapter 1 students. Project hours, for instance 
was not found to be related with achievement gains. It is, however, noted 
that findings from some of the previous studies indicated that mere length 
of the school day or class period «ioes not necessarily- influence student 
achievement. The positive relationship depends on how the availifele time 
was used, not just the amount of time available (Staliings, 3<»B0/ Karweit 

and SlajKin, 1981) . 

Project enrollment correlated negatively with achievement gains^. 

Similarly, school enrollment was found to be inversely related to 
achievement gains. The negative correlations appear generally consisteftt 
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with Pokey's (1976) finding that larger district enrollment tended to show 
poorer average achievement. Contrary to Polley's results, absenteeism was 
found to be related to student achievement, at least for one of the three 
school years. 

Previous findings (Glasman and Biniaminov, 1981) relating to 
expenditures receive support from the present study. Polley (1976) found a 
positive relationship between median teacher salaries and student 
achievement. Moreover, per pupil cost on principals' salaries was also 
positively related to achievement. The major share (over 80%) of Chapter 1 
expenditures consists of personnel costs. The per pupil cost index used in 
the present study was highly similar to expenditure indices used in 
Polley's study. Cost indices used in the present study correlated 
positively with achievement gains. 

Pretest achievement status and school type correlated negatively with 
achievement. The finding suggests that lower achieving projects tended to 
produce greater gains and that students in lower grades tended to make 
greater gains. It is noted that in other recent Chapter 1 studies (e.g., 
national annual Title 1 evaluation reports) similar trends were found. 
That is, the lower achieving students show more of a "spurt" of growth 
while higher achieving students show steady, but less dramatic gains. This 
occurs not as a result of the regression to the mean but rather as a 
consequence of the development process. Developmental factors may also 
account for the fact that students in lower grade levels tended to make 
greater gains. 

The priWy areas of interest in the present study consist of such 
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raanipulable program features as project setting and instructional 
approach. Contrary to negative perceptions expressed by some researchers 
(Harnischfeger, 1980), the pull-out setting was shown to be superior to 
other project settings (e.g., regular classroom) examined in the study. 
Even when grade level effects were taken into account, the overall trend 
still appeared tp favor the pull-out setting. The trend was less clear 
with respect to the effects of different instructional approaches. In a 
majority of instances, howeveif, the ctMobination approach was shown to have 
worked as well as, if not better than other approaches. The assertion of 
superiority for the pull-out setting and, to a lesser extent, for the 
combination instructional approach was made on the basis of replicability 
(i.e., that the same pattern of results was observed for different school 
years and within each school year in different grade levels) rather than on 
the basis of statistical significance. As many researchers (e.g.. Coats, 
1970; Cronbach, 1975? Carver, 1978) have pointed out, statistical 
significance, while important, should not be used as the sole criterion for 
interpreting results of educational research. Replicability is a far more 
useful and appropriate criterion. 

The study suggests several potent variables for predicting Chapter 1 
student achievement. These variables include school type, school/proDect . 
enrollment, pretest achievement status, per pupil cost, absenteeism, 
project setting and instructional ^iift^roach. Kesults of the study support 
the following conclusions: 

1. Project impact as measured by NCE gains over the three school 

years covered by the study has remained quite stable ranging from 
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7 NCEs to 8 NCEs for the reading projects and hovering around 11 
NCEs for the math projects. 

Students in the lower grade levels can be expected to make greater 
NCE gains than students in higher grade levels. The difference is 
probably a result of the developmental process rather than a 
consequence of differential program effectiveness. 
Size of project and/or school enrollment is inverstly related to 
achievement gains. This suggests that small projects located, in 
smeCll schools seem more conducive to learning^basic skills than 
large: projects located in large schools. 

Students who scored l^r on the pretest tend to make greater 
achievement gains than their higher scoring counterpirM- This, 
again, is probably due to developoiental factprs rather than 
program effectiveness. 

Per pupil cost is positively related to achievement gains. 
Undoubtedly, this occurs because per pupil cost is a surrogate for 
program elements (e.g., more experienced teaching staff, better 
facilities and materials) which tend to produce high achievwnent. 
There is some evidence that absenteeism is inversely related to 
achievement gains. This finding is generally in congruence with 
results of recent time-on-task studies. 

In terms of achievement gain, students who received Chapter 1 
instruction in the pull-out setting are likely to perform as well 
as, if not better than their counterparts in other settings. This 
suggests that despite its apparent drawbacks (e.g., loss of 



regular classroom instruction, transition tiaf) the pull-out 
setting remains a viable option for providing services to 
Chapter 1 youngsters. 
8. Students who received Chapter 1 instruction under a combination of 
instructional jqpproachea (e.g., a prescribed system supplemented 
by miscellaneous materials) are likely to make as much, if not 
greater, achievement gains" as their counterparts Under other 
single approaches. This perhaps point,s to the validity of 
allowing the teaching staff flexibility in using materials in ways 
most suitable for- individual students within a project. 
During the course of the study, both the state and district 
compensatory educatiort staff have indicated the inqportance of teacher 
variables as contributing factors to project impact. It was pointed out 
that the regular classroom (intervention) setting had worked 'well in some 

/ . 4 

V . 

schools where Chapter 1 teachers were well trained and where communication 

[ ■ / 

between the regular classroom teacher and Chapter 1 staff was effective. 

Furthermore, low staff turnover was perceived to be a major contributing 

factor to project inqpact. These teacher variables are undoubtedly 

important elements of an effective project and should receive attention in 

future studies of project impact. 

\ 
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Table 1 

List of Variables Pertaining to Project Profiles 



Variable # 1 District code 

2 School code 

3 School type 

4 Grade levels covered by school 

5 Project setting 
4 Subject area 

7 Instructional approach 

8 Student grouping 

9 Diagnostic testing 

10 Recordkeeping- ^ 

11 Grade levels served by project 

12 Inservice training (Orientation) 

13 Inservice training (Planning/subject area) 

14 inservice training (Management) 

15 Inservice training (Evaluation) 

16 Project ^nrollnent 

17 Days of project operation 

18 Total hours of instruction 

19 Minutes of instruction per week 

20 Project, staff (in PTEs) ^ 
^2l School enrollment 

22 Percent of welfare families 

23 School staff (in PTEs) 

24 Nuiiii>er of PAG members 

25 Number of PAG members who were I'itle I parents 

26 Number of students tested 

27 Pretest NGE 

28 Posttest'NCB 

29 NGE gain 

30 Percent of students making positive gains 

31 Number of days absent 

32 Project funding 

33 Per pupil cost 
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Table 2 

» ^ 

Comparison of Project Setting Effects for 
Reading Projects of 1978-79 School Year (N-112) 



Variable 



Regular 
Classroom 



PullHDut 



Combination 



No. of projects 


13 


53 


9 


Pre I4CE 


22.31 


16.63 


21.18 


Per Pupil Cost 


433.00 


781.07 


773.55 


NCE Gain 


5.91 


8.73 


7.64 


Adjusted NCE Gain* 






J 



^ ^ 

*F (2, 70) ■ .34; no adjusted means were calculated. 

Note: Projects do not sum to total N because of missing data. Projects for 
which no relevant information was available were excluded from the 
analysis. 
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Table 3 



Comparison of Project Setting Effects for 
Reading Projects of 1979-80 School Year (N»118) 



Regular 

variable Classroom Pull-Out Combination 

No. of Projects 7 92 10 

^re NCE 20.07 19.49 23.78* 

Per Pupil Cost 479.86 732.60 595.30 



NCS Gain 

Adjusted NCE Gain* 



5.08 8.30 6.18 



*F (2r 104) ■ .91? no adjusted means. were calculated. 

Notes Projects do not sum to total N because of missing dat^ Projects 
for which no relevant information was available were excluded from 
the analysis. 
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Table 4 
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Comparison of Project Setting Effects for 
Reading Projects of 1980-81 School Year (N«119) 



Regular 

Variable Classroom Pull-Out Combination 



No. Of Projects 31 62 10 



Pre NC£ 



24.34 21.37 23.06 



Per Pupil Cost 584.81 625.95 581.90 

NCE Gain 6.78 ^ 9.73 8.65 

Adjusted NCE Gain* 



*P (2y 98) - .85; no adjusted means were calculated. 

Note: Projects do not sum to total N because of missing data. Projects for 
which no relevant information was available were excluded from the 
analysis. 
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Table 5 



CoBparison of Project Setting Effects for 
Mathematics Projects of 1978-81 School Years 

(N - 57) 



Variable 



Regular Classroom Pull-Out 



No. of Projects 


14 


Pre NCE 


15.13 


Per Pupil Cost 


420.07 


NCE Gain 


8.36 


Adjusted NCE Gain* 


9.09 



12 

23.39 
519.00 
13.33 
12.47 



♦Adjusted for pretest and per pupil cost differences. P (1, 22) - 2.43, 
P > .05 

Note: Projects do no sum to total N because of missing data. Projects 

for which no relevant information was available were excluded from 
the analysis. 
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Table 6 



Comparison of Instructional Approach Effects for 
Readi^ng Projects of 1978-79 School Year (N-112) 



Variable 


Instructional 
Systems 


Instructional 
Materials Only 


Combination 


Others 


No. of Projects 


6 


3 


19 


60 


Pipe NCE 


22.4 


24.00 


18.41 


17.73 


Per Pupil Cost 


- 532.17 


701.66 


• 726.26 


723.25 


NCE Gain 


8.32 


4.83 


10.39 


7.31 


Adjusted NCE Gain* 


10.86 


6.34 


10.25 





♦Adjusted for pretest and per pupil cost differences. P (3, 82) - 2.57, P>.05 

Note: Projects do not sum to total N because of missing data. Projects 
for which no relevant information was available were excluded from 
the analysis. 
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Table 7 



Comparison of Instructional Approach Effects for 
Reading Projects of 1979-80 Scttbol Year (N-118) 



Variable ^ 


Instructional 
Systems 


Instructional 
Materials Only 


' Comb inaction 


Others 


Nrt-^ of Proiects 


10 


3 


61 


35 


Pre NC£ 


16.42 


25.93 


19.11 


21.26 


Per Pupil Cost 


738.60 


535.67 


709.05 


699.14 


NCE Gain 


6.50 


5i53 


' 9.3-7 


6.16 


Adjusted NCE Gain* 


5.29 


8.08 


9.15 




♦Adjusted for pretest 


and per pupil cost differences. 


P (3, 103) « 


3.63, 



p < .05 

Note: Projects do not sua to total N because of aissing data. Projects 
for which no relevant inforaation was available were excluded froa 
the analysis. 
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Table 8 



Coaparison of Instructional Approach Effects for 
Reading Projects of 1980-81 School Year (N-118) 



Variable 



Instructional 
Systens 



No. of Projects 7 

Pre NCE 18.76 

Per Pupil Cost 533.28 

NCE Gain 10.07 

Adjusted NCE Gain* 9.32 



Instructional 

Materials Only Combination Others 



4 

21.67 
682.00 
10.05 
9.22 



56 

22.62 
606.18 
9.23 
9.22 



17 

23.89 
585.94 
6.25 
6.79 



*Adjusted for pretest and per pupil cost differences. P (3, 78) ■ 1.03, 
p > .05 

Note: projects do not sum to total N becaupe of missing data. Projects 
for which no relevant information was available were excluded from 
the analysis. 
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Table 9 

Comparison of instructional J^proah Effects for 
Mathematics Projects of 1978-81 School Year 

(N - 57) J 

Variable Combination Others 

No. of Projects 
Pre NCE 

Per Pupil Cost 
NCE Gain 

Adjusted NCE Gain* 



*P (1, 20) - .004; no adjusted means were calculated. 

Note: Projects do not sum to total N because of missing data. Projects 
for which no relevant information was available were excluded from 
the analysis. 



11 

23.56 
447.73 
10.32 



13 

25.19 
46a.61 
10^56 
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